Some systems that provide augmented reality (AR) environments do so by generating a view that includes both an image of a physical environment (e.g., captured using a video camera function on a device) and one or more objects or other AR features that are added to the image of the physical environment. Observing such an AR environment can give the user the realistic impression of being in and/or traveling through a world that is a mixture of physical and AR objects. The interaction between the user and these objects can be an important aspect of the user's experience of the AR environment.